…ocal:

Address review on PR scitix#9 (two correctness issues in the local: handler):

- local: now copies bundled files via a .partial temp + atomic replace,
  with cleanup on error. An interrupted copy (SIGKILL/ENOSPC) no longer
  leaves a truncated file that is_downloaded reports as ready. Mirrors
  the url: handler's staging contract.

- The basename collision guard previously scanned only url: sources, but
  local: stages to the same <dest>/<name>/<basename> layout. Rename
  _validate_url_basenames_unique -> _validate_staged_basenames_unique and
  scan both url: and local:, so url/url, local/local, and url/local
  collisions are all rejected at @sieval_dataset registration.

Tests: add local/local and url/local collision cases to test_meta.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ocal:

Address review on PR scitix#9 (two correctness issues in the local: handler):

- local: now copies bundled files via a .partial temp + atomic replace,
  with cleanup on error. An interrupted copy (SIGKILL/ENOSPC) no longer
  leaves a truncated file that is_downloaded reports as ready. Mirrors
  the url: handler's staging contract.

- The basename collision guard previously scanned only url: sources, but
  local: stages to the same <dest>/<name>/<basename> layout. Rename
  _validate_url_basenames_unique -> _validate_staged_basenames_unique and
  scan both url: and local:, so url/url, local/local, and url/local
  collisions are all rejected at @sieval_dataset registration.

Tests: add local/local and url/local collision cases to test_meta.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ocal:

Address review on PR scitix#9 (two correctness issues in the local: handler):

- local: now copies bundled files via a .partial temp + atomic replace,
  with cleanup on error. An interrupted copy (SIGKILL/ENOSPC) no longer
  leaves a truncated file that is_downloaded reports as ready. Mirrors
  the url: handler's staging contract.

- The basename collision guard previously scanned only url: sources, but
  local: stages to the same <dest>/<name>/<basename> layout. Rename
  _validate_url_basenames_unique -> _validate_staged_basenames_unique and
  scan both url: and local:, so url/url, local/local, and url/local
  collisions are all rejected at @sieval_dataset registration.

Tests: add local/local and url/local collision cases to test_meta.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dataset): compare on-the-wire bytes so gzip responses aren't flagged as truncated downloads

The truncation check in url.py compared bytes *written to disk* against
Content-Length. That breaks for Content-Encoding responses: when the
server sends gzip, Content-Length is the compressed size while
iter_bytes() yields the larger decompressed body, so written bytes
always exceed Content-Length and the download is falsely rejected.

Compare r.num_bytes_downloaded (raw on-the-wire bytes, before httpx
decompresses) against Content-Length instead — the two are now measured
on the same, compressed scale, so gzip-served files pass while genuinely
truncated transfers still fail.

Regression: SQuAD's train-v2.0.json is gzip-served; Content-Length=9551051
but the decoded body is ~42MB, which the old written-bytes check rejected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(datasets): add local: downloader for package-bundled dataset files

feat(datasets): add LocalHandler to stage local:<relpath> assets from sieval/datasets/_data/ into dest_root/<dataset_name>/
feat(datasets): enforce normalized package-relative paths and reject absolute / traversal inputs in _bundled_path()
feat(datasets): align output filename behavior with URL downloader via shared url_path_basename fallback (download)
test(datasets): cover scheme validation, traversal guards, copy/skip/force behavior, and is_downloaded() checks for local downloader
Co-authored-by: Claude Sonnet 4.6 (Anthropic) noreply@anthropic.com

* feat(datasets): subpackage auto-discovery + vendored RULER assets

Extend the lazy-loading discovery in datasets/__init__ (and the stub sync)
to scan subpackages, so a benchmark can live in its own directory. Vendor
RULER's prompt templates and string_match metrics under community/ruler,
and bundle the Paul Graham Essays haystack corpus (read via the local:
scheme) with a one-shot regeneration script.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(datasets): RULER synthetic datasets (niah/qa/vt/cwe/fwe)

Add the five parameterized RULER loaders covering all 13 task configs.
Synthesis is aligned to upstream NVIDIA RULER: binary-search context
sizing (niah/vt/cwe), VT's built-in 1-shot ICL, FWE's tokens_to_generate
budget reservation, and essay-repeat on overflow. vt/cwe/fwe are fully
synthetic (no download); niah/qa stage from bundled/url sources.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(tasks): RULER tasks (chat + base-gen) and effective-length report

Add 10 RULER tasks — chat and base/completion variants of niah/vt/cwe/
fwe/qa — over shared scoring (string_match_all/part) and endpoint mixins.
Add `sieval leaderboard ruler-effective` to aggregate a multi-length sweep
into per-length 13-task averages and the threshold-based effective length.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(examples): RULER eval configs and multi-length sweep generator

Add single-length (chat) and QA-only (base/completion) example configs,
plus a generated multi-length sweep and the gen_ruler_sweep.py tool that
expands the 13 configs across length tiers with per-tier YARN factors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(examples): inject tokenizer_model into generated RULER sweep

Add --tokenizer-model to gen_ruler_sweep.py (default = --checkpoint) and
write it into every dataset's args, so synthesis sizes prompts with the
evaluated model's own tokenizer — matching upstream RULER, which sets the
dataset tokenizer to the model path. Without this, prompts were measured
with the gpt-4 tiktoken default and the length tiers drifted off-target.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(datasets): load hotpotqa from HF instead of the defunct CMU URL

curtis.ml.cmu.edu is no longer reachable. Replace the url: source with
hf:hotpotqa/hotpot_qa and rewrite _read_hotpotqa to call load_dataset()
with the HF distractor split. Context is now read from the HF schema
(context={'title': [...], 'sentences': [[...], ...]}) rather than the
JSON file's list-of-pairs format. Update tests to mock load_dataset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(examples): correct hotpotqa dataset path in RULER sweep generator

- Update qa_hotpotqa subdir from 'ruler_qa' to 'hotpotqa'
- Align generated YAML configs to actual dataset locations:
  - SQuAD: ${SIEVAL_DATA_DIR}/ruler_qa
  - HotpotQA: ${SIEVAL_DATA_DIR}/hotpotqa
- Add SGLang deterministic inference flag (enable_deterministic_output)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(ruler): remove base_gen endpoint support

- Remove completion endpoint (base_gen) tasks
- Supported endpoint now chat-only via ChatModel
- Rationale: RULER evaluation focuses on instruction-following via chat templates

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* refactor(ruler): simplify to chat-only endpoint

- Update _base.py: remove _BaseGenBase class
- Gen script: remove endpoint parameter, only support chat
- Update docstrings and tests
- Simplify task class registry to remove endpoint branching

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler-qa): handle hotpotqa config fallback and path correction

- Add error handling in _read_hotpotqa: fallback from 'distractor' to 'fullwiki' config
- Fix BuilderConfig 'distractor' not found error with graceful fallback
- Regenerate qwen3-8b_4k_sglang.yaml with correct hotpotqa path

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler-qa): correct hotpotqa loading strategy

- Remove fullwiki fallback, keep distractor-only
- Handle local vs remote dataset loading: try with config first, then without
- Fixes 'BuilderConfig' errors when loading from cache or local files

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* refactor(ruler): align scoring with community.ruler metrics

- Import string_match_all and string_match_part from community.ruler.eval.constants
- Unify recall and QA scoring: collect predictions+references in feedback, compute batch score in report
- Align RecallFeedback with QaFeedback structure (prediction + references)
- Ensures metrics match upstream RULER exactly across all task types

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(ruler): support task ordering from config file

- Add extract_task_order() to parse YAML config and extract task ordering
- Update collect_sweep() signature to accept optional task_order parameter
- Update summarize() to include task_order in output when provided
- Add --config parameter to ruler-effective command
- Enables output sorted by config definition order instead of alphabetical

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(ruler): include detailed per-task results in output

- Add collect_sweep_with_tasks() to group scores by task (not just aggregate)
- Update summarize() to include 'per_task' section with individual task scores
- Per-task results are ordered according to config task_order when available
- Output structure: per_task[length]['tasks'][task_base_name] = score

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(output): display per-task scores in ruler-effective text renderer

- Add per-task details section to text output
- Tasks are displayed in config order (if task_order available) or alphabetically
- Output format:
    Task Details:
      4k
        ruler_niah_single_1: 92.50
        ruler_niah_single_2: 89.30
        ...

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler-gen): add SGLang deterministic inference to server overrides

- Add enable_deterministic_inference: true to model overrides (server-side)
- This is different from extra_body parameter (client-side)
- SGLang server startup now receives this parameter via overrides
- Fixes log showing enable_deterministic_inference=False

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(output): match task_order names with task_base names in per-task display

- task_order contains full names with _Nk suffix (ruler_niah_single_1_4k)
- tasks dict keys are base names without suffix (ruler_niah_single_1)
- Strip suffix from task_order entries before matching against tasks dict
- Fixes missing task details in terminal output

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(ruler): auto-detect effective_config.yaml for task ordering

- If --config not provided, automatically search for effective_config.yaml
- Search in the first output directory (where results are stored)
- This file is persisted by sieval run and contains the original config
- Falls back to no task_order if file not found
- Enables seamless task ordering without explicit config parameter

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler): strip suffix from task_order before matching with task base names

- task_order contains full names with _Nk suffix (ruler_niah_single_1_4k)
- tasks_at_length keys are base names without suffix (ruler_niah_single_1)
- Strip suffix from task_order_bases before matching in summarize()
- Fixes issue where per_task results were always empty

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(ruler): add Qwen3 8B SGLang configurations

- Add support for multiple context lengths (8k, 64k, 128k)
- Add base configuration for Qwen3 8B with SGLang
- Update dependencies and Python version constraint to 3.12.*
- Lock dependency versions for stability

* refactor(ruler): rework RULER datasets and tasks to reproduce model continuation

Align the RULER datasets and tasks with the upstream NVIDIA RULER
implementation (prepare.py + synthetic scripts) so the prompt ends with
the answer cue and the model continues from it, and fix latent bugs that
broke loading, scoring, and downloads.

Datasets (sieval/datasets/ruler):
- ruler_vt.py: fix the port — keyword-only signatures, binary-search
  noise sizing, essay/noise haystacks, built-in 1-shot ICL; drop the
  broken bare `from _common import` for the package-relative import
- ruler_niah.py: import _build_haystack/_NEEDLE from _common instead of a
  duplicated local copy; add the missing `from nltk.tokenize import
  sent_tokenize` that broke the essay path
- _common.py: become the single source of _build_haystack and the
  haystack constants; drop the stray `self` parameter
- ruler_cwe.py: fix the broken os.path.dirname() call; load the optional
  english_words.json from the staged data dir; replace bare `except`
- ruler_qa.py: switch tokenizer.encode() to text_to_tokens()
- ruler_{niah,vt,qa,cwe,fwe}.py: append answer_prefix to the template
  before generation (mirrors prepare.py) so the loader can split it back
  out; align every *DatasetSample TypedDict with the real rows.append
  schema (index/input/outputs/length/answer_prefix; niah adds
  token_position_answer); default the tokenizer to openai/cl100k_base so
  loading is offline and portable
- ruler_cwe.py: register english_words.json as a `url:` source so it is
  fetched by `sieval dataset download`

Tasks (sieval/tasks/ruler/_base.py):
- preprocess: split the prompt into a user turn (body) and an assistant
  prefill turn (answer_prefix) so the model continues the cue; the prefill
  toggle (continue_final_message/add_generation_prompt) is left to the run
  config's extra_body, not hardcoded, to avoid clobbering it
- RulerRecallSample and the recall scoring mixin now read `outputs`
  (previously `answer`, which never matched the dataset rows)

Community (sieval/community/ruler):
- add scripts/tokenizer.py with select_tokenizer (hf/openai backends)
- remove the obsolete scripts/template.py; trim datasets/eval constants

Config & meta:
- examples/qwen3-8b_4k_sglang.yaml: enable continue_final_message /
  add_generation_prompt in the model extra_body
- sieval/meta/index.json: regenerate from the updated registries

Tests (tests/unit/{datasets,tasks}/ruler):
- rewrite the dataset and task unit tests for the new row schema and the
  user+assistant message structure; use the offline tiktoken tokenizer;
  fix stale imports (build_tokenizer) and signatures (_get_example). All
  36 ruler unit tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): adapt dataset generation for Qwen3 with model-aware tokenization

* Use HuggingFace tokenizer (model-specific) instead of tiktoken for all RULER
  datasets (NIAH, VT, CWE, FWE, QA) to match actual token counting at inference
  time, fixing context-length overages with Qwen models
* Account for Qwen3 thinking tags (<think>...</think>) overhead in dataset
  synthesis by pre-calculating token cost when enable_thinking=false
* Add enable_thinking parameter to all dataset loaders to track extended thinking
  overhead during prompt sizing
* Update gen_ruler_qwen3_8b_sglang.py to pass tokenizer_type='hf' and
  tokenizer_path (model path) instead of tokenizer_model
* Add Qwen3-aware prompt preprocessing to inject thinking tags for extended
  reasoning even when enable_thinking=false (workaround for thinking framework)
* Fix HotpotQA data loading: add more_context field from doc pool to support
  distractor sampling in QA generation

Fixes context-length violations in RULER evaluation where dataset synthesis
used tiktoken (generic) but inference used model-specific tokenizers with
different token counts. Qwen3's thinking tags now properly accounted for.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* chore(examples): restore infer-recipe-override.yaml

Re-add the example config that was inadvertently removed during the
Qwen3 ruler dataset-generation work.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* build(deps): restore project metadata and add ruler dependency group

The pyproject metadata had been overwritten with PDM template defaults
(description, license Apache-2.0→MIT, requires-python, build-system
removed). Restore it from the canonical version and add only the intended
`ruler` optional-dependency group (tiktoken, wonderwords, numpy, scipy).
Relock with the correct >=3.12,<3.15 target so no existing pins drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ruler): centralize thinking-tag prefill into a config-driven helper

The Qwen3 `<think></think>` placeholder was hardcoded in two places — the
dataset loaders (reserving token budget) and the task base (prefilling the
assistant turn) — and the task gated on a brittle `"qwen" in name` sniff.
The two sites also disagreed on the default (loader treated unset as False,
task as None).

Introduce `thinking_prefill(model_name, enable_thinking)` in
datasets/ruler/_common.py as the single source of truth, re-exported from
the package. Both loaders and the task base now consume it, so writers only
set `enable_thinking: false` in config and the string mapping lives in one
model-aware place. Non-Qwen3 / default paths return "" — zero impact on the
general case.

Also fixes 5 pre-existing test failures (preprocess reads self.model, which
the bare __new__ instances lacked) by giving them a non-reasoning stub, and
adds branch coverage for the helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): correct VT/CWE shot count to n_shot=1 (kshot, not 0-shot)

RULER's VT and CWE prompts each embed one in-context demonstration —
upstream defaults to `num_fewshot=1` (CWE) and always prepends an ICL
example (VT). Labeling them `n_shot=0` was wrong both factually and for
anomaly routing, which synthesizes a `zero_shot`/`few_shot` tag from n_shot.

Rename to the `kshot` convention (mirroring `drop_kshot_gen`):
  ruler_{vt,cwe}_0shot_gen → ruler_{vt,cwe}_kshot_gen
  Ruler{Vt,Cwe}ZeroShotGenTask → Ruler{Vt,Cwe}FewShotGenTask
  n_shot=0 → 1, display_name → "(few-shot, generative)"

Regenerated stubs + meta index; updated the __init__ docstring, the sweep
generator's class map, and tests. Dropped the orphaned
test_gen_ruler_sweep.py (its target script was renamed away long ago).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(ruler): restore NVIDIA license headers and upstream attribution

The vendored RULER files (datasets/eval constants, tokenizer) had blank
`# adapted from` lines and were missing their upstream Apache-2.0 headers.
Add back the NVIDIA copyright/license preambles and point each file at its
exact upstream source in NVIDIA/RULER, satisfying the community/ license
attribution requirement. Also drops dead code in tokenizer.py (commented-out
tenacity import, empty GeminiTokenizer stub).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ruler): satisfy ruff — drop dead code and fix lint

Ruff cleanup across the RULER CLI and sweep generator:
- remove the unused `task_order` parameter from collect_sweep /
  collect_sweep_with_tasks (no call site ever passed it; docstring promised
  ordering the body never did) and the unused `model_type` local
- collapse a nested `if` and delete commented-out deterministic-inference code
- rename the unused per-task loop variable to `_length_val`
- wrap long lines and correct two help strings that named the old
  gen_ruler_sweep.py (now gen_ruler_qwen3_8b_sglang.py)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): satisfy ty — type the vendored TASKS table at the call sites

community/ is excluded from type checking and its TASKS dict mixes int/str
values, so ty inferred `int | str` at every loader call site (breaking
subscription, concatenation, and int parameter defaults — ~26 diagnostics).

- add a typed `ruler_task()` accessor + `RulerTaskSpec` TypedDict in
  datasets/ruler/_common.py as the single typed view over TASKS; migrate all
  five loaders onto it and drop their direct TASKS imports
- fix a latent niah bug surfaced by typing: the join wrapped split() in an
  extra list, which would TypeError when remove_newline_tab=True (untested)
- narrow `test_set` (HFDataset | None) with asserts in dataset tests; pass a
  typed `_SELF: Any` for the self=None unbound-method calls in task tests
- add a ty override for gen_paul_graham_essays.py's intentionally-unvendored
  html2text/bs4 imports

ty now passes clean (was 50 diagnostics); ruff clean; 2052 tests pass at
98.49% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(leaderboard): remove the ruler-effective command

Drop the RULER effective-length report and align the leaderboard CLI back
with upstream main: delete leaderboard/ruler.py and its test, the
ruler-effective command, and the _render_text_ruler_effective renderer plus
its registration. Update the sweep generator's docstring/header to point at
`leaderboard report` instead of the removed command.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): pin the HotpotQA HF revision for reproducibility

RULER QA's distractor documents come from hotpotqa/hotpot_qa; pin the source
to a fixed commit (@1908d6af…) so `sieval dataset download` fetches a stable
snapshot instead of whatever HF `main` points to, matching the gsm8k dataset's
revision-pin pattern. Regenerated meta/index.json.

The two url: sources (SQuAD, english_words.json) still need sha256 checksums,
but that decorator field lands with upstream's checksum work — deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(examples): drop the stray run.sh dev script

run.sh hardcoded a personal /root/sieval environment and referenced a config
(qwen3-8b_8k_sglang.yaml) that isn't in examples/ — it was a local test
runner committed by accident, not a usable example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(ruler): drop the qwen3-8b sweep generator from the PR

gen_ruler_qwen3_8b_sglang.py is an experiment-only tool for producing a
specific Qwen3-8B/SGLang multi-length sweep config — it's not part of the
RULER benchmark itself and nothing in the package, tests, or examples imports
it. Kept on the experiment/ruler-sweep-gen branch for reproducing runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(ruler): correct attribution to NVIDIA RULER and pin reference SHAs

The implementation was attributed to OpenCompass, but a code comparison shows
the synthesis logic (binary-search haystack sizing, sent_tokenize, num_fewshot
loop, randle_words fallback, ceiling-division doc repeat, Zipfian generation)
is ported from NVIDIA RULER — OpenCompass is a simplified downstream port that
drops all of these. Fix the attribution accordingly:

- task reference_impl: source opencompass → NVIDIA/RULER, url repointed at
  RULER's scoring file (scripts/eval/synthetic/constants.py — what the thin
  task actually mirrors), with notes clarifying synthesis lives in the dataset
  loader. URLs pinned to a commit SHA (meta-index sync rejects mutable refs).
- add module docstrings to the five dataset loaders naming the RULER synthesis
  file each is ported from, with the required AI-Generated marker (they had
  neither before).
- pin the community/ vendored-file attribution URLs to the same SHA.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): add leaderboard ruler-avg for the 13-subtask headline mean

RULER's headline score is the unweighted mean of its 13 subtasks, but each
subtask runs as an independent sieval task with its own report.json — the
per-task `leaderboard report` matrix isn't that number, and computing it by
hand isn't reproducible. Add a minimal `leaderboard ruler-avg` command that
collapses the ruler_* runs into the per-length mean (and an overall mean),
with a text renderer. Pure aggregation lives in _ruler_avg.py for testing.

Deliberately minimal: just the mean, no thresholds or effective-length logic
(unlike the earlier ruler-effective command). parse_length guards against
mistaking a variant index (the `_1` in ruler_niah_single_1) for a 1-byte
context length — only `<n>k` or bare numbers >= 1024 count as lengths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): round ruler-avg means to one decimal

Round the per-length and overall averages to 1 decimal place in the
aggregation layer, so both the text and JSON output are consistent and match
the precision used when transcribing scores into spreadsheets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ruler): drop fixed-width padding on ruler-avg per-length value

Use {:.1f} instead of {:6.1f} for the per-length average — the right-aligned
6-char pad isn't needed and reads cleaner.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(downloaders): add AI-Generated marker to test_local

Sync the local: downloader test with the standalone feat/local-downloader
branch — the new test file needs the AI-Generated marker per CONTRIBUTING.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ruler): ruff-format _ruler_avg

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(ruler): fix last opencompass reference in niah task docstring

The niah task docstring still said it mirrored OpenCompass's evaluator; the
scoring actually mirrors NVIDIA RULER's string_match_all (vendored in
community/ruler/eval). Align the wording with the other ruler tasks and fix a
stray ". ." typo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(datasets): make local: copy atomic and extend basename guard to local:

Address review on PR #9 (two correctness issues in the local: handler):

- local: now copies bundled files via a .partial temp + atomic replace,
  with cleanup on error. An interrupted copy (SIGKILL/ENOSPC) no longer
  leaves a truncated file that is_downloaded reports as ready. Mirrors
  the url: handler's staging contract.

- The basename collision guard previously scanned only url: sources, but
  local: stages to the same <dest>/<name>/<basename> layout. Rename
  _validate_url_basenames_unique -> _validate_staged_basenames_unique and
  scan both url: and local:, so url/url, local/local, and url/local
  collisions are all rejected at @sieval_dataset registration.

Tests: add local/local and url/local collision cases to test_meta.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ruler): unify 5+5 dataset/task classes into one RulerDataset + RulerZeroShotGenTask

Replace five separate Dataset classes (niah/vt/cwe/fwe/qa) and five Task
classes with a single RulerDataset (subtask= param) and RulerZeroShotGenTask.
report() now groups by (context_length, subtask) internally, removing the
ruler-avg CLI command.

Dataset internals split into datasets/ruler/ subpackage:
- ruler.py: RulerDataset + RulerDatasetSample + _stamp
- _shared.py: cross-subtask constants + helpers (thinking_prefill, _len_tag)
- _niah.py / _vt.py / _cwe.py / _fwe.py / _qa.py: per-subtask synthesis

Public import path (sieval.datasets.ruler) is unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(ruler): support subpackage dataset discovery and fix thinking prefill

- fix(datasets/__init__.py): extend lazy-loading registry to discover exports
  from dataset subpackages (e.g., RulerDataset from ruler/). Add _iter_subpackage_dirs()
  and scan subpackage modules in _discover_dataset_exports().

- refactor(scripts/sync_package_stubs.py): extract _discover_dataset_classes()
  helper and mirror discover_tasks() pattern to support both flat modules and
  subpackages. Generates __init__.pyi with proper imports for subpackage exports.

- fix(datasets/ruler/ruler.py): when subtask="all", map each subtask to correct
  data directory (ruler_niah, ruler_cwe, ruler_qa, etc.) so load() finds the right
  files regardless of input path. Fixes FileNotFoundError when loading all 13
  subtasks together.

- fix(datasets/ruler/_shared.py): fix thinking_prefill() to emit "<think>\n" when
  enable_thinking=True, not empty string. Ensures Qwen3 actually generates reasoning
  content instead of empty/malformed blocks. Clarify docstring and branching logic.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: pre-commit formatting and code quality checks

- fix(ruff): reformat code to fix line length violations (88 char limit)
  across ruler dataset/task modules and CLI utils.

- fix(ruff): simplify nested if statements using logical operators (SIM102)
  in _shared.py thinking_prefill().

- refactor(datasets/ruler): promote _len_tag to public API (len_tag) to
  support production use in ruler_0shot_gen task report generation. Update
  __init__.py exports and all call sites.

- fix(ruff): mark unused function parameters with underscore prefix
  (_name_or_path in _fwe.py, _haystack in _niah.py) to suppress ARG001.

- chore: delete gen_ruler_qwen3_8b_sglang_thinking.py (temporary generation
  script scheduled for removal).

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler): pre-commit formatting and docs for long-context eval config

## Pre-commit Compliance

- reformat code to fix line length violations (88 char limit)
  across ruler dataset/task modules and CLI utils.

- simplify nested if in _shared.py thinking_prefill() using
  logical operators (SIM102).

- mark unused function parameters with underscore prefix
  (_name_or_path in _fwe.py, _haystack in _niah.py) to suppress ARG001.

- promote _len_tag to public API (len_tag) for production use
  in ruler_0shot_gen task report generation. Update __init__.py
  exports and all call sites.

## Documentation & Examples

- enhance example RULER eval configs with critical prerequisites:
  * continue_final_message: true + add_generation_prompt: false REQUIRED
    (without them, answer-prefix continuation breaks silently)
  * tokenizer_path MUST match target model (RULER sizes prompts per-tokenizer)
  * YaRN config for >32k context lengths

- update qwen3-8b_64k_sglang.yaml and qwen3-8b_8k_sglang.yaml
  to match ruler-multilength.yaml prerequisite documentation

## Cleanup

- delete gen_ruler_qwen3_8b_sglang_thinking.py (temporary generation
  script scheduled for removal)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* refactor(ruler): drop assistant prefill, move answer_prefix to user turn, add think_budget

- Replace `thinking_prefill()` with `tokens_to_generate(task, enable_thinking, think_budget)`:
  when thinking is enabled, generation budget = think_budget + base answer tokens.
- All dataset loaders (_niah, _vt, _cwe, _fwe, _qa) accept `think_budget: int = 0`
  and remove the `thinking_overhead` term from length calculations.
- `ruler.py` threads `think_budget` through all 13 subtask dispatch paths.
- `ruler_0shot_gen.preprocess` emits a single user turn `input + answer_prefix`
  instead of `[user, assistant(prefill)]`; no more `continue_final_message` dependency.
- Tests updated: `thinking_prefill` parametrize → `tokens_to_generate` coverage;
  preprocess test asserts single-turn user message.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* refactor(ruler): add Qwen3 thinking tag overhead and multi-subtask loading

- Add QWEN3_THINKING_TAG_OVERHEAD (4 tokens) for <think>..</think> tags
- Enhance tokens_to_generate() with model_name param to handle model-specific
  thinking overhead: Qwen3 includes 4-token tag, other models omit it
- Pass model_name through all 5 dataset loaders (_niah, _vt, _cwe, _fwe, _qa)
- Support RulerDataset.load(subtask=[...]) to load multiple subtasks and
  concatenate into one dataset with auto path resolution
- Fix unused parameter in _niah.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(ruler): add dual message pattern support for assistant vs user prefill

This change enables compatibility between two message formatting patterns
for reasoning models, particularly supporting both feat/ruler and feat/ruler_exp branches.

Key changes:

- Add thinking_prefill() helper in _shared.py to handle model-specific thinking
  tag generation with Qwen3-specific logic:
  * When thinking enabled: return empty string (model continues in <think> block)
  * When thinking disabled: return "<think>\n\n</think>\n\n" (empty block to skip)
  * Other models: always return empty string

- Update RulerZeroShotGenTask.preprocess() to detect and support both patterns:
  * Assistant-message pattern (feat/ruler): prefilled assistant turn with thinking
    placeholder + answer_prefix, detected via continue_final_message flag
  * User-message pattern (feat/ruler_exp): answer_prefix appended to user message

- Export thinking_prefill from ruler.__init__.py for downstream use

- Restructure test_ruler_0shot_gen.py for clearer test organization

- Add new unified test suite in test_ruler_unified.py (239 lines)

Maintains backward compatibility while supporting both message patterns.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* docs: clarify extra_body config detection logic for assistant prefill pattern

* docs(examples): add comprehensive comments to all ruler YAML configs

- Add detailed header comments to 4 ruler YAML configs (in English):
  * Purpose and evaluation scenario for each configuration
  * Key configuration features and their differences
  * Setup steps, editable fields, and expected outputs
  * Performance expectations and warnings where applicable

- Fix: rename ruler-qwen3-8b-nonthinking-withyarn-64k.yaml.yaml → .yaml

* fix(dataset): reword url size-mismatch error; pin gzip test to RULER's dev-v2.0.json source

The byte-count guard rejects any num_bytes_downloaded != Content-Length
mismatch, including an over-read, so "truncated download" was a misnomer
for that direction — reword to "size mismatch".

Also align test_download_accepts_compressed_response's docstring to the
file RULER actually fetches (dev-v2.0.json: Content-Length=800683,
decoded ~4.4MB) instead of train-v2.0.json, so the Regression note
describes the real download path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(tasks): shorten comment lines to satisfy ruff line-length check

Reduce verbosity of comments in ruler_0shot_gen.py to meet 88-char limit:
- Simplify detection logic description
- Abbreviate assistant-pattern explanation
- Keep semantic meaning intact

* feat(tasks): add HMMT Feb 2025 and IMO-AnswerBench benchmarks (#22)

* feat(tasks): add MathArena-aligned HMMT Feb 2025 benchmark

Add HMMT February 2025 (30 problems) final-answer math benchmark, aligned
to the MathArena reference impl (eth-sri/matharena @a11194de
configs/competitions/hmmt/hmmt_feb_2025.yaml): boxed prompt + last-boxed
extraction (reuses the vendored sieval/community/matharena.py), equivalence
via math-verify — same pipeline as hmmt_feb_2026.

Dataset pinned to hf:MathArena/hmmt_feb_2025@6fdc4277...; sieval dataset
download hmmt_feb_2025 resolves (30 rows).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(tasks): add IMO-AnswerBench benchmark (Google DeepMind IMO-Bench)

Add IMO-AnswerBench (400 short-answer olympiad problems from the DeepMind
IMO-Bench suite, hf:Hwilner/imo-answerbench, pinned). Grading is vendored from
the upstream answer_verification.py @66b014f1 (community/imo_bench.py): math-verify
equivalence with a normalized-string fallback when either side can't parse — NOT
an LLM judge. The reference harness is agentic (tool-call answer, "reason step by
step" prompt); for a non-agentic generative run we add a boxed answer format and
reuse the matharena last-boxed extractor. math-verify stays lazy-imported.

sieval dataset download imo_answer_bench resolves (400 rows); ruff/ty/preflight
green; unit tests cover the verifier (integer / LaTeX-equiv / string fallback)
and import discipline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(imo-answerbench): gen-mode answer normalization for grading

The vendored official verify_math_answer assumes the agentic harness's clean
tool-submitted answer. In our non-agentic generative run the model boxes verbose
answers ($-wrapped, \left/\right, function-prefixed, multi-answer "A or B",
trailing newline), which math_verify.parse mis-parses -> correct answers scored
wrong. Re-grading the DeepSeek-V4-Pro run showed 26/310 non-truncated answers were
mis-graded (71.6% -> 80.0%), all pure formatting / clean multi-answer differences.

Add verify_answer_gen alongside the verbatim verify_math_answer: fast-path is the
official grader; only on failure does it normalize (strip $/\left\right/\text{}/
whitespace, "and"/"or"->comma) and set-match multi-answer golds. Deliberately
identity-only per split item (no per-item math_verify) — math_verify.parse grabs a
coincidental number out of expression answers (e.g. "n=3k"->3), which caused false
positives; whole-answer math equivalence still uses the official fast path. Net:
26 recovered, 0 regressions, 0 false positives on the run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(meta): regenerate index.json after rebase onto #8 (HF pins)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(imo-answerbench): mark experimental + document divergences; empty-normalize guard

Address review on PR #22 (IMO-AnswerBench is a non-strict port of upstream
IMO-Bench, so classification/provenance must be explicit in shipped metadata):

- status="experimental" (was defaulting to "stable" — a frozen leaderboard
  contract): the harness type changes (agentic tool-call submission -> generative
  \boxed{} extraction) and there is a non-upstream scoring layer.
- Enumerate every upstream divergence in reference_impl.notes + module docstring:
  agentic->generative; added boxed prompt + separator vs bare "reason step by
  step"; verify_answer_gen normalizer contributes ~11% (raw verify_math_answer =
  260/400 = 65.0% -> 73.25%); HF mirror Hwilner/imo-answerbench vs upstream
  answerbench.csv; dual lineage (prompt/extraction = matharena, grader = IMO-Bench);
  \boxed{} format-compliance confound (known limitation); infer prereqs
  (large max_tokens + generous read-timeout; budget-sensitive score).
- Fix latent empty-normalize false positive in verify_answer_gen: _atom_equiv now
  returns False when either side is empty after normalization (e.g. \text{No} vs
  \text{Yes} both reduce to "" — upstream returns False). Dormant on pinned data
  (0/400 \text-wrapped golds; score unchanged at 293/400) but grades less strictly
  than upstream otherwise; regression test added.

HMMT Feb 2025 stays status="stable" (faithful matharena clone, no bespoke scoring).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: peter-scitix <peter-scitix@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): make feat/ruler pass CI (lint, types, preflight, tests)

Bring the branch to green across the pre-commit, typecheck, and checks jobs:

- deps: relock pdm.lock (content hash went stale after the main merge)
- types: RulerDataset.load gets a `subtask=None` default + runtime guard so the
  override no longer violates LSP; assert `test_set is not None` in the ruler
  unit tests; ty-ignore unresolved-import for the ruler deps group
  (tiktoken/wonderwords), matching the existing t_eval override
- datasets: add sha256 checksums for the two RULER url: sources
  (english_words.json, dev-v2.0.json) now that upstream checksum enforcement
  landed; regenerate sieval/meta/index.json
- examples: fix stale task class RulerCweZeroShotGenTask -> RulerZeroShotGenTask;
  repoint the broken examples/README.md link to ruler-qwen3-8b-nonthinking.yaml
- tests: mock LocalHandler in the download-all iteration test (ruler is the
  first dataset with a local: source); drop duplicate test_ruler_unified.py
  (its async tests used @pytest.mark.asyncio, unsupported in this project)
- lint: ruff/format + whitespace fixes across ruler files and examples

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(downloaders): correct local scheme docstring to match generate-not-commit flow

The corpus behind a local: source is produced out-of-band by a generation
script (e.g. scripts/gen_paul_graham_essays.py) into sieval/datasets/_data/ and
is not committed to the repo — the user runs the generator themselves. Reword
the module docstring and data-root comment which previously described the file
as "generated once and committed inside the package".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ruler): true bring-your-own local: corpus (+ nltk dep, example fix)

Three related cleanups from the #11 review:

local: → true BYO. The Paul Graham essay corpus is not redistributable
(Apache-2.0 covers NVIDIA's scraper, not the essays) and must not ship in the
wheel or repo. Replace the copy-from-package model:
- LocalHandler.download() moves no bytes — no-op if the corpus is already
  staged, else raises LocalSourceUnavailable naming the expected path. Drops the
  _bundled_path/_data resolution entirely.
- `sieval dataset download` catches that and prints the instructions as a
  non-fatal notice, so the fetchable url:/hf: sources still download and the
  command doesn't hard-fail on the BYO piece.
- gen_paul_graham_essays.py takes --data-dir (default $SIEVAL_DATA_DIR) and
  writes straight to <data-dir>/ruler/PaulGrahamEssays.json.gz — the loader's
  read path — so no staging copy is needed.
- Rewrite test_local.py for the no-op/raise semantics.

nltk dependency. niah/vt essay tokenization imports nltk, but the ruler extra
only declared tiktoken/wonderwords/numpy/scipy; check_dep_coverage passed only
because nltk was reachable via ifeval, so a clean `pip install sieval[ruler]`
broke at load. Relock adds only nltk's group membership — no version drift.

Example fix. The nonthinking-withyarn-64k config ran enable_thinking:false on
the model but enable_thinking:true in the dataset args, over-reserving the
thinking token budget for a run that never thinks. Align the dataset arg.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): consistency and robustness improvements per maintainer review

- Retry loop consistency: remove `else: break` guards from _niah.py and _cwe.py
  to match upstream RULER exactly (niah/qa/vt/cwe allow unbounded retry).

- Download robustness: track missing local: (BYO corpus) sources and emit
  clear final summary with exit code 1, so missing corpus is not mistaken for
  success. Non-fatal per-source notices remain for individual missing pieces.

- Docstring accuracy: clarify that only the assistant-message pattern
  (continue_final_message: true) is validated against NVIDIA/RULER ab17b78;
  user-message append mode is supported for thinking compatibility but not
  empirically verified.

Co-Authored-By: Claude Sonnet 5 (Anthropic) <noreply@anthropic.com>

* refactor(ruler): onboarding + encapsulation nits

- Docstring clarity: gen dependencies (html2text, beautifulsoup4, certifi)
  now documented inline in pyproject.toml [project.optional-dependencies.ruler]
  comment; no new optional group to avoid bloat.

- Architecture: Inline _ChatGenBase into RulerZeroShotGenTask (single subclass).
  Replace private self.model._kwargs/_model access with public meta() API.

- Robustness: Add annotations to wonderwords private API calls (_get_words_from_text_file)
  in _cwe.py and _niah.py; document version pinning requirement.

- Pre-commit: Remove --maxkb=2000 bump (bundled corpus now via local: bring-your-own,
  no longer exceeds 500KB default).

- Config cleanup: Narrow ty overrides (remove gen_paul_graham_essays.py exemption,
  ruler remains for CI light env). Remove scripts/ T201 exemption (not needed).

- Examples: Merge YaRN 128K variant (ruler-qwen3-8b-nonthinking-withyarn-*.yaml)
  into nonthinking.yaml comments; keep only baseline (nonthinking) + thinking.

Co-Authored-By: Claude Sonnet 5 (Anthropic) <noreply@anthropic.com>

* fix(ruler): resolve script and type-checker issues in final nits

- Add noqa: T201 annotations to print statements in scripts/, which are
  manual tools (gen_paul_graham_essays.py, check_layer_imports.py, check_preflight.py).
  These are not runtime code and print is the intended output mechanism.

- Narrow tool.ty.overrides to specific files instead of glob patterns:
  * ruler loader files (_niah.py, _cwe.py) + test_ruler.py: unresolved-import
    (wonderwords, tiktoken absent from CI light env, intentional)
  * gen_paul_graham_essays.py: unresolved-import (html2text, beautifulsoup4,
    certifi are doc-generation deps, not runtime)

- All preflight/lint checks now pass (ty, ruff, tests, preflight).

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* feat(mbpp): add dataset and few-shot base-model task (#12)

* feat(mbpp): add dataset and few-shot base-model task

* fix(mbpp): remove score refs from task docstring

* fix(mbpp): use HF full-config dataset, simplify task args

* style(mbpp): apply ruff-format to test file

* test(mbpp): restore loader test; annotate timeout deviation

* fix(mbpp): align param naming and timeout with task family

* feat(openbookqa): add dataset and k-shot generative task (#19)

* feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate (#21)

* feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate

sglang's OpenAI /v1/completions rejects echo=True + logprobs, blocking
PPL-style tasks (ARC, HellaSwag, ...) on the sglang backend. SglangGenModel
overrides only _alogprobs_impl to call sglang's native /generate
(return_logprob=True + logprob_start_len=0), returning per-token logprobs
over the full echoed input sequence. Normalizes byte-level BPE token text
so extract_option_logprob matches reliably. Dispatched via a new
model-config field `engine: sglang` in the leaderboard session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(models): rebase SglangGenModel on Model[str], parse top_logprobs, fix review nits

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(models): guard against sglang radix-cache logprob truncation; review nits

* refactor(models): address review — keep prompt out of request_params, guard sglang derived downgrade + missing prompt_tokens, validate generate response

* docs(models): note cross-engine default generation-length divergence

* fix(models): coalesce duplicate top-logprob tokens; fail loud on missing token text

Two review follow-ups on the sglang /generate backend:

- _parse_top_logprobs: distinct token ids can normalize to identical text
  (byte-level "ĠA" and a literal " A" both -> " A"). Coalesce by keeping
  the highest logprob so a low-probability duplicate can't clobber the real
  one, matching CMMLU's max-over-strip semantics.
- _normalize_token_text: a server launched with --skip-tokenizer-init ignores
  return_text_in_logprobs and returns None token text. Raise an actionable
  error instead of crashing on None.replace or silently degrading every token
  to "".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): validate model 'engine' field in dry-run config check

_setup_models rejects a bad engine value, engine on a chat model, and engine
on a derived model, but run_dry_run only does static validation and never
builds models -- so these slipped through `sieval eval --dry-run` and only
failed mid-run. Mirror the statically-decidable guards into _validate_models
so dry-run catches them, matching how 'type' is already validated. The
engine-on-non-gen guard uses the inferred type at runtime, so statically we
flag only the explicit 'type: chat' case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Ethan <ethan@scitix.ai>

* feat(gsm8k): add DeepSeek-Math-aligned 0-shot chat-model task (#29)

* feat(gsm8k): DeepSeek-Math-aligned 0-shot chat task

* fix(gsm8k): full-set accuracy denominator + accurate community docstrings

* feat(ifbench): add dataset and few-shot base-model task (#13)

* feat(ifbench): add dataset and few-shot base-model task

* fix(ifbench): address review — collapse loader to pinned HF source, doc/build fixes

* docs(ifbench): list IFBench in README benchmarks and install extras

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(ifbench): fix reproduction recipe in docstring/notes, correct README count

* chore(ifbench): sync meta index after rebase

* fix(ifbench): mark task experimental; note eval-time NLTK fetch

The official temperature=0 protocol does not reproduce on Qwen3 thinking
mode, and the 37.3 match is a stochastic sample under substituted sampling.
The port is faithful but the reproduction is unverified, so set
status="experimental" rather than the default "stable".

Also document in the task docstring that scoring lazily fetches NLTK corpora
on first use (pre-baked in Docker; offline runs pre-stage via
SIEVAL_IFBENCH_NLTK_DATA).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ethan <ethan@scitix.ai>

* feat(tasks): add clp eval mode + naming category (#23)

* feat(tasks): add clp eval mode + naming category

* fix(cli): include clp in --eval-mode help text

The `EvalMode.CLP` value was added to the enum but the `task list
--eval-mode` help still enumerated only (gen/ppl). The filter itself is
generic, so this is a docs/UX consistency fix, not a behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Ethan <ethan@scitix.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(imo-answerbench): normalize in extraction + verbatim grader, promote to stable (#28)

* refactor(imo-answerbench): normalize in extraction + verbatim grader; promote to stable (RFC #26)

Move answer normalization to the parsing layer and grade 100% with the vendored
upstream verify_math_answer — removing the bespoke scoring layer that kept the task
experimental, and fixing format false-negatives.

- Delete bespoke verify_answer_gen / _atom_equiv / hand-rolled set-matching.
- Add normalize_answer (parsing only): strip $, ^\circ, \left/\right/spacing, a
  leading f(x)= prefix and a TRAILING \text{… for/where/such/with …} qualifier;
  rewrite \{…\} to a set math_verify parses. Conservative — inline \text (e.g.
  piecewise "\text{ if } x>=2") is kept.
- Task grades with verify_math_answer on normalized inputs, symmetric $-wrapping
  like the HMMT sibling (verify(parse(f"${gold}$"), parse(f"${pred}$"))), so
  math_verify does all equivalence (commutativity / factoring / set-equality).
- Promote status experimental -> stable; grading is now verbatim upstream, only
  parsing is bespoke (legitimate; every gen task has one).

Re-graded the DeepSeek-V4-Pro run (stored predictions): 293 -> 307/400 = 76.75%
(target >=76%); +17 genuine math-equalities recovered, 0 false positives. 3 prose
answers old bespoke matching happened to accept are now left ungraded (out of scope:
prose/infinite-sets/quantified families need the agentic answer channel or an LLM
judge — noted). verify_math_answer untouched (verbatim upstream). HMMT 2025 unaffected.

Refs #26.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(imo-answerbench): repin to official v2 data + honest attribution

- Data: hf:Hwilner/imo-answerbench (deprecated v1) -> official
  answerbench_v2.csv (google-deepmind/superhuman/imobench, pinned commit +
  sha256). v2 (2026-02-12) fixes ambiguous statements / wrong answers (11
  golds + 26 statements changed vs v1). Two v2 rows carry upstream
  spreadsheet artifacts (algebra-036, geometry-004) kept verbatim + noted.
- Attribution: reference_impl now names the authoritative source
  (google-deepmind/superhuman + imobench.github.io + arXiv 2511.01846);
  grader stays vendored verbatim from the third-party EnvCommons/IMO-Bench,
  byte-identical to the math_verify-only official scoring, byte-stable
  pin->HEAD. Grader logic unchanged.
- Baseline: real sieval eval on v2, DeepSeek-V4-Pro pass@1 = 317/400 =
  79.25% (max_tokens=131072, 0 truncation, transient API fails resumed).
  The v1 76.75% was an offline re-grade; relabeled reference-only.
- Add CSV-loader dataset unit test locking the v2 pin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(imo-answerbench): reframe grader as deterministic substitute (review)

Per @ethan-scitix review: the official AnswerBench grader is an LLM autograder
(AnswerAutoGrader, Gemini 2.5 Pro; arXiv 2511.01846 §2.3/§5.1) and the paper
rejects symbolic/SymPy matching as too narrow; the 'No LLM graders / math_verify'
wording is EnvCommons's README, not the paper. Official imobench/ ships data only.
Reframe notes + docstrings: the vendored EnvCommons math_verify grader is a
deliberate, reproducible, strictly-more-conservative SUBSTITUTE for the official
LLM autograder (under-counts vs official), not a reproduction of it. Nit:
'byte-identical' -> 'behaviorally identical' (imports made lazy). No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: peter-scitix <peter-scitix@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: release 0.6.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(pyproject.toml): format with taplo

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* chore: regenerate stubs and meta index with ruler

- sync_package_stubs.py: regenerate .pyi exports with RulerDataset/RulerZeroShotGenTask
- sync_meta_index.py: update sieval/meta/index.json to include ruler task/dataset

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* chore(pdm): update lock file

- pdm lock --update-reuse to sync with pyproject.toml

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(ruler): add transformers dep, restore scripts T201 exemption, annotate port divergences

deps: pull `transformers` into the `ruler` optional group. Both example configs
use tokenizer_type: hf, triggering `from transformers import AutoTokenizer` in
community/ruler/scripts/tokenizer.py, so a clean `pip install sieval[ruler]` hit
ImportError — masked only by t-eval's transitive transformers. Relocked with
--update-reuse (group membership only, no version drift).

lint: restore the `scripts/**/*.py` T201 exemption in per-file-ignores and drop
the per-line `# noqa: T201` it had forced onto check_layer_imports /
check_preflight / sync_meta_index / gen_paul_graham_essays.

docs: annotate three reviewed RULER-port divergences (comments only, no behavior
change):
- ruler_0shot_gen.infer(): upstream caps generation per subtask; one class
  serving 13 subtasks can't express per-subtask caps via a single infer_args.
- _qa.py shrink loop: narrow `except AssertionError` vs siblings' `except
  Exception`; documents the unreachable over-shrink ValueError path.
- ruler-qwen3-8b-thinking.yaml: dataset think_budget (0) vs serving
  thinking_budget (8192); fits only because context_length is 2x max_seq_length.
Also drop a stale feat/ruler_exp compatibility note from thinking_prefill's
docstring in _shared.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(pyproject.toml): format T201 exemption array with taplo

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): resolve reviewed port issues (else-break, divergence notes, aggregation test)

Address the outstanding RULER-port review items:

- _vt.py: drop the lone `else: break` from the size-shrink retry loop so all
  four subtask generators (niah/qa/vt/cwe) allow unbounded retry consistently,
  completing the intent of 4b3f8db6.
- _vt.py: pass `remove_newline_tab` through to the ICL example generation
  instead of hardcoding False. Upstream applies the flag to the ICL example;
  its token count feeds `_binary_search_noises`, so under the non-default
  `remove_newline_tab=True` the hardcoded False shifted `num_noises` and
  diverged from NVIDIA/RULER (dormant at the default, fidelity-only fix).
- ruler_0shot_gen.py: enumerate the per-subtask generation-cap divergence in
  ReferenceImpl.notes so `status="stable"` satisfies the non-strict-repro rule
  (scoring was already noted); regenerate meta/index.json.
- leaderboard/commands.py: inline the single-caller `_resolve_run_models`
  helper back into `report()` (leftover extraction from the removed ruler-avg
  command) and drop the now-unused RunInfo import.
- test_ruler_0shot_gen.py: add discriminating report() aggregation tests
  (mean-of-means vs flat mean, QA part-match routing, fails counting, empty
  finals).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): fix single-subtask path crash + QA guard, enumerate divergences

- Centralize the subtask->data-dir mapping (`_subtask_data_path`) so a direct
  single-subtask load applies the `/ruler` staging subdir. Previously only the
  all/list branches appended it, so `subtask="niah_single_2"` (any essay/squad
  subtask) hit FileNotFoundError. Also collapses the duplicated all/list path
  dicts into one loop.
- QA retry loop: widen `except AssertionError` -> `except Exception` (matches
  the _niah/_cwe/_vt siblings) and fail loud at the floor instead of letting
  `random.sample(..., negative)` escape or spinning forever, so
  max_seq_length < 4096 raises a clear error. Fix the wrong "unreachable in
  practice" comment.
- reference_impl.notes: enumerate every divergence (uncapped max_tokens,
  HotpotQA doc order, essay concat order, CWE word cap) with why each is
  score-neutral; scope the stable contract to <=32k for CWE.
- gen_paul_graham_essays.py: correct the false "bytes match upstream" comment
  (this script concatenates in URL-list order; upstream groups repo-then-html).
- Add single-subtask path regression tests (_subtask_data_path + loader routing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): scrub personal paths from example configs, fix presence_penalty placement

- Replace hardcoded `/root/models/...` checkpoint and tokenizer_path with
  `/path/to/...` placeholders in both example configs so nothing machine-specific
  ships; users supply their own paths.
- Move `presence_penalty` from `extra_body` up to the top-level sampling params
  (nonthinking config), where the standard sampling knobs live.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): align HotpotQA doc order to upstream (alphabetical)

Experimental QA-only change on top of feat/ruler (re-run the Qwen3-8B repro
before promoting, since it changes generated bytes):

- HotpotQA document order: switch from first-seen insertion order to
  alphabetical `sorted(set())`, byte-identical to upstream qa.py so the document
  indexing and distractor selection match upstream. Removes the ordering
  divergence.
- reference_impl.notes: drop the now-resolved HotpotQA-order divergence; the
  generation cap, essay concat order, and CWE <=32k scoping remain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): read staged HotpotQA copy instead of always fetching online

qa_hotpotqa was routed to the base data dir, so _read_hotpotqa called
load_dataset("<data_dir>", "distractor", ...) — the base dir is not a
distractor-config dataset, so it always raised and silently fell back to
an online fetch. The pinned staged download was dead code, and eval failed
under HF_HUB_OFFLINE=1 despite a completed download.

Point the loader at the HF mirror's real landing spot <data_dir>/hotpotqa/
hotpot_qa (snapshot_download -> dest_root / repo_id), keeping the online
fetch as an explicit last resort. Extract _HOTPOTQA_REPO_ID as a single
source of truth shared by the loader path, the online fallback, and the
@sieval_dataset source so the download target and read path can't drift.

Add regression tests covering staged-copy-first reads and the
staged-absent online fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ruler): tidy model-name resolution loop and doc-set comprehension

- commands.py: expand model-name resolution into an explicit loop with
  typed RunInfo list for readability
- _qa.py: fold HotpotQA doc collection into a set comprehension
- ruler.py: note english_words.json is staged for parity but not
  load-bearing in the <=32k stable scope

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ruler): align HotpotQA doc order + cap generation per subtask

Experimental changes layered on feat/ruler, split out per review (the Qwen3-8B
repro should be re-run on this branch before promoting, since both change
generated bytes):

- HotpotQA document order: switch from first-seen insertion order to
  alphabetical sorted(set()), byte-identical to upstream qa.py. Removes the
  ordering divergence entirely (distractor selection now matches upstream).
- Per-subtask generation cap: stamp each sample with `gen_budget`
  (tokens_to_generate for its RULER task, incl. any thinking overhead) and pass
  it as max_tokens in infer(), matching upstream's per-task cap
  (128/30/120/50/32). Replaces the implicit context-window bound that only held
  when serving context == max_seq_length.
- reference_impl.notes: drop the now-resolved generation-cap and
  HotpotQA-order divergences; only the essay concat order and CWE <=32k scoping
  remain.
- Tests: gen_budget stamping (_stamp + fwe schema) and the infer() max_tokens cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): account for message-template overhead in prompt sizing

RULER prompt fitting counted only raw content + generation budget, never
the inference-time message-template overhead (role markers, prefilled
<think></think> block). At long contexts this let wrapped prompts overflow
max_seq_length, producing failed samples (e.g. 128k runs).

Introduce a two-function split in _shared.py with no double-counting:
- calculate_prompt_tokens(): input side only — wraps the prompt in the real
  qwen3 template (via community template.py) and counts tokens using the
  RULER text_to_tokens interface. No think_budget.
- tokens_to_generate(): output side (max_tokens) — includes think_budget and
  the generated <think> tag overhead, since thinking is generated.

The only possibly-overlapping token, the <think> tag, is counted exactly
once: prefilled in the prompt for non-thinking (input side), generated for
thinking (output side).

Thread enable_thinking/model_name through all five subtask fitters and
per-sample length checks (_niah/_qa/_cwe/_fwe/_vt); VT keeps the ICL fragment
as a raw count to avoid double-wrapping. FWE reserves template overhead in
input_max_len and raises a clear error if think_budget swallows the context.
Fix a SyntaxError (missing comma) in community/ruler/datasets/template.py.

Tests: 64 pass; end-to-end FWE at 4k/32k in both thinking modes stays within
max_seq_length with gen_budget correctly including think_budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): resolve qwen3 template key case-insensitively (fixes overflow)

get_template looked up lowercase "qwen3-nonthinking"/"qwen3-thinking" but
template.py registers them capitalized ("Qwen3-nonthinking"). The mismatch
silently fell back to the base template ("{task_template}"), so
calculate_prompt_tokens counted ZERO message-template overhead (~13 tokens:
role markers + prefilled <think></think> block). Prompts were sized that
much too large and overran max_seq_length by a few tokens at inference
(e.g. non-thinking 128k: 130947 input + 128 = 131075 > 131072).

- Resolve template keys case-insensitively; fail loud (KeyError) if a Qwen3
  template is genuinely missing instead of silently degrading to base.
- Restore the Qwen3-thinking template (had lost its /think marker and
  assistant turn).

With the fix, sizing matches the model's real chat_template within 1 token
(conservative). Adds regression tests: non-zero qwen3 overhead,
case-insensitive resolution, and fail-loud on a missing key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): align prompt-token reserve with upstream model_template_token

Match NVIDIA/RULER prepare.py + niah.py (commit ab17b78): reserve
model_template_token = len(text_to_tokens(RAW template)), where the template
string still contains the literal {task_template} placeholder. Upstream does
`max_seq_length -= model_template_token` then fits with `<=`; folding the
reserve into calculate_prompt_tokens (content_tokens + model_template_token)
is algebraically identical.

Previously we tokenized the template with the real content substituted in,
reserving the *exact* overhead with zero slack. That let a sample fill the
context to exactly max_seq_length (input + completion == 131072), which the
serving engine rejects as exceeding the limit. Counting the unformatted
template reserves the placeholder's tokens too (~4-5) — they are replaced by
real content at inference and never emitted, giving the same headroom upstream
relies on. Verified: sizing now exceeds the model's real chat_template input
count by ~5 tokens (conservative), and FWE at 32k stays within budget in both
thinking modes.

Tests updated: base/unknown model reserves the "{task_template}" token count
(upstream-consistent); qwen3 thinking template distinguished by absence of a
prefilled </think> block rather than a /think marker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ruler): drop /think marker from Qwen3-thinking template

Qwen3 reasons by default when the assistant turn is opened without a
prefilled <think></think> block, so the explicit /think soft-switch is
redundant. The thinking template now just opens the assistant turn; the
non-thinking template still prefills the empty block to suppress reasoning.
get_template distinguishes the two by presence/absence of the prefilled block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ruler): own Qwen3 templates locally + reserve template tokens once

Addresses two review findings:

Structure:P0 — community/ vendor fidelity. The Qwen3-nonthinking/Qwen3-thinking
entries do not exist in upstream RULER's template.py at the pinned SHA, so
having them in the vendored community/ file polluted its provenance (and a
re-vendor would silently drop them). Move the two Qwen3 templates into a
sieval-owned _QWEN3_TEMPLATES dict in datasets/ruler/_shared.py and restore
community/ruler/datasets/template.py to the upstream-verbatim key set.
get_template now resolves Qwen3 from the local dict; other models still use the
vendored Templates (case-insensitive, base fallback).

Nit:P1 — compute sample-invariant work once. Replac…